Automatic and Manual Metrics for Operational Translation Evaluation Workshop Programme

This paper presents a study on human and automatic evaluations of translations in a French-German translation learner corpus. The aim of the paper is to shed light on the differences between MT evaluation scores and approaches to translation evaluation rooted in a closely related discipline, namely translation studies. We illustrate the factors contributing to the human evaluation of translations, opposing these factors to the results of automatic evaluation metrics, such as BLEU and Meteor. By means of a qualitative analysis of human translations we highlight the concept of legitimate variation and attempt to reveal weaknesses of automatic evaluation metrics. We also aim at showing that translation studies provide sophisticated concepts for translation quality estimation and error annotation which the automatic evaluation scores do not yet cover.